We investigate the dependence of the amount of the observed galactic cosmic ray (GCR) influx on the solar North-South asymmetry using the neutron count rates obtained from four stations and sunspot data in archives spanning six solar cycles from 1953 to 2008. We find that the observed GCR influxes at Moscow, Kiel, Climax and Huancayo stations are more suppressed when the solar activity in the southern hemisphere is dominant compared with when the solar activity in the northern hemisphere is dominant. Its reduction rates at four stations are all larger than those of the suppression due to other factors including the solar polarity effect on the GCR influx. We perform the student's t-test to see how significant these suppressions are. It is found that suppressions due to the solar North-South asymmetry as well as the solar polarity are significant and yet the suppressions associated with the former are larger and more significant.